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This paper describes the verification and evaluation procedures 
followed in the development of new features for the Traffic Service 
Position System (tsps). Beginning with the definition of new feature 
requirements, these procedures are adhered to throughout the tsps 
development cycle. Hardware and software designs are reviewed to 
verify that all requirements are met and to ensure that Bell System 
standards for reliability are maintained. Finally, both system labo- 
ratory and site testing are performed to verify the proper implemen- 
tation of each new feature and to evaluate the overall performance of 
the tsps. 

I. INTRODUCTION 

A significant part of the effort required for any switching system 
development involves determining whether the system performs 
properly. The ultimate judge of system performance is the user. In 
making this judgment, the user considers both system reliability and 
maintainability. Since Bell System standards for service are very high, 
measures of acceptable performance are rigorous. Thorough plans are 
made to ensure that switching systems provide this required high level 
of performance. These plans begin with the initial concept of a switch- 
ing system or new feature and continue throughout the development 
process. Every design decision is based on providing the best possible 
service at the lowest possible cost. Each decision is evaluated for its 
impact on the customer who uses the switching system and on the 
operating company that must administer and maintain it. 

This paper discusses the methods used to verify and evaluate system 
performance of the Traffic Service Position System (tsps). It also 
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describes the facilities specifically designed to support the verification 
and evaluation process. 

II. STAGES OF THE TSPS VERIFICATION AND EVALUATION PROCESS 

The verification and evaluation process for new tsps features in- 
volves all aspects of system development. This process begins with the 
determination of feature objectives and continues with the verification 
that the design requirements for each new feature are consistent with 
these objectives. Finally, the system is tested to ensure that the 
implementation reflects the design requirements. This process is di- 
vided into several stages. The stages of verification and evaluation 
discussed in this article are (i) requirements reviews, (ii) design 
reviews, (Hi) circuit analysis or program reviews, (iv) unit testing, (v) 
integration testing, and (vi) system testing. Figure 1 shows these six 
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Fig. 1 — Six stages in the TSPS verification and evaluation process. 
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stages and how the results of each stage are used in subsequent stages 
of the development. 

The first stage of the verification and evaluation process begins 
when a new tsps feature or system capability is proposed. Each new 
feature is analyzed to determine the service and maintenance require- 
ments that must be satisfied. This analysis can be very extensive for 
some new features, such as the Automated Coin Toll Service (acts) 
feature. To perform this analysis for acts, an in-service tsps was 
modified to simulate acts for a comprehensive human factors trial. 1 
The results of this trial guided the formulation of the requirements for 
acts. These requirements are documented in development require- 
ments memoranda and form the basis for detailed design and devel- 
opment. 

The detailed design and development of a new feature is generally 
accomplished by partitioning the feature into functional units which 
are then assigned to development teams for implementation. Contin- 
ued division of responsibility within a team for the development of a 
functional unit is dependent on the complexity of that unit. This 
partitioning applies to both hardware and software portions of the 
system. 

The second stage of the tsps verification and evaluation process 
begins when design reviews are held to determine if the conceptual 
aspects of each functional unit are consistent with the overall system 
requirements. Once the architectural design of the functional unit has 
been evaluated in this manner, the development team begins the 
detailed design of the functional unit. The design of each element in 
the functional unit is reviewed. Depending on the number and com- 
plexity of functional unit interfaces, members of other tsps develop- 
ment teams may participate at this point in the process. After this 
review, development specifications memoranda and circuit descrip- 
tions are usually written documenting the detailed design. 

The hardware development proceeds with the generation of circuit 
schematic diagrams and the specification of the circuit components 
used in constructing the circuit. The layout of the circuit components 
and their interconnections are specified, and prototype circuits are 
built. The prototype circuits are tested off-line from the system to 
ensure that they perform as required. Manufacturing tests are also 
generated for digital circuits using a logic simulation program. 2 After 
these tests and prototypes of the digital circuits are verified with a 
circuit pack tester, testing begins in the system laboratory. Once the 
circuit is verified to operate in accordance with the design intent, 
manufacturing and factory test information is transmitted to Western 
Electric. This activity is scheduled so that standard Western Electric- 
supplied hardware will be available when site testing begins. 
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The software development is done in parallel with the hardware 
development. After each program is written, a program review is 
conducted (i) to ensure that all development requirements are met; 
(ii) to verify that the interfaces between programs are correct; and (Hi) 
to check that each program instruction is correct. Problems identified 
and corrected during this program review require much less effort than 
if they were left to be resolved during later stages of the verification 
and evaluation process. 

The fourth stage in this process is the formulation of functional tests 
which utilize special test faculties (see Section III) and which ensure 
that each functional unit meets the design requirements. Upon the 
completion of unit testing, the fifth stage of the verification and 
evaluation process occurs when integration testing is performed. In- 
tegration tests ensure that each functional unit performs in a total 
system environment. Specifically, interactions between functional 
units and hardware-software interfaces are emphasized. Once the 
integration tests have been successfully run, the functional unit designs 
are considered frozen, in that changes to a functional unit can only be 
made on a more formal and controlled basis. The frozen functional 
units are now ready for system testing, the sixth and last stage in the 
verification and evaluation process. 

During system testing, additional functional tests are written, and 
the total set of functional tests are performed by an independent group 
to ensure that any biases held by the design engineers are not reflected 
in the interpretation of test results. Beginning with this phase of tsps 
testing, every change introduced into the system — hardware or soft- 
ware — must be initiated by a trouble report which specifies the seri- 
ousness of the problem so that corrections can be generated on a 
priority basis. Changes must be submitted in a formal manner and 
approved by a special committee called the change review committee. 

Beginning with the system testing phase of the verification and 
evaluation process, each change is tested incrementally. That is, the 
correction to each problem, when possible, is considered an independ- 
ent entity. Each correction can be rejected if not deemed appropriate 
by either the change review committee or the system testers. This 
procedure is used (i) to provide a high degree of visibility to all changes 
being introduced into the system, (ii) to provide a procedure whereby 
each change is individually and thoroughly tested, and (Hi) to provide 
a procedure whereby design engineers can make changes without 
repeatedly going through the integration process. 

To this end, each change submitted by the design engineer must be 
accompanied by a very specific test procedure which has been verified 
before the change is submitted. System testers then take great care to 
ensure that the test procedure is appropriate and that the change does 
not invalidate a previously verified functional capability. In some 
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cases, this can only be done by repeating a long series of previously 
completed functional tests. This verification and evaluation process is 
greatly enhanced by a comprehensive set of test facilities discussed in 
the following section. These test facilities are available to both design- 
ers and system testers. 

III. SYSTEM LABORATORY TESTING 

tsps verification and evaluation is done at Indian Hill using two 
system test facilities. Each test facility contains a tsps and its associ- 
ated peripheral subsystems (i.e., a Position Subsystem No. I, 3 Position 
Subsystem No. 2, 4 Remote Trunk Arrangement, 4 Station Signaling 
and Announcement Subsystem 5 — see Fig. 2). Minicomputers and var- 
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Fig. 2 — Traffic Service Position System No. 1. 
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ious microprocessor-controlled circuits are utilized to produce the 
desired hardware and software environment. 6 In this environment, the 
system laboratory user can control and monitor the operation of the 
tsps. Figure 3 shows one of the tsps system laboratories at Indian 
Hill. 

One aspect of controlling the tsps operation is to establish the 
desired hardware and software configuration. The minicomputer pro- 
vides three major capabilities for reconfiguring or modifying the sys- 
tem. First, the minicomputer provides the capability to load the tsps 
with a new program, with the system's program stores being automat- 
ically reconfigured to match the memory requirements of the new 
program. Second, portions of the tsps program can be modified with 
changes assembled by the minicomputer. This capability is used ex- 
tensively in the incremental testing of modifications to frozen pro- 
grams. Third, automatic fault insertion by the minicomputer is used 
to exercise system diagnostics in the preparation of trouble locating 
manuals. These manuals are used in conjunction with diagnostic 
outputs to identify hardware faults. The automatic process by which 
these faults are inserted helps to increase trouble locating manual 
resolution by allowing efficient generation of a large number of sample 
diagnostic results. 

Once the desired hardware and software configuration has been 
established, the minicomputer can then be used to monitor the oper- 
ation of the tsps. Data associated with a specific event or combination 
of events can be recorded and later retrieved by the minicomputer. 
When correlated with the normal system responses such as teletype- 
writer messages, those data can be used to resolve problems or to 
confirm the proper operation of the tsps. 

An important part of testing tsps features is ensuring that system 
interactions involving customers and operators are correct. Many 
inputs processed by a tsps result from these interactions. To provide 
similar inputs in the system laboratory, stimuli comparable to those 
generated by customers and operators are produced using system 
utilities. A microprocessor-controlled facility is provided to simulta- 
neously generate calls from many trunks. Both the traffic-handling 
characteristics of each trunk and call types can be changed under 
program control. Facilities are also provided to simulate the local and 
toll offices associated with a single tsps trunk, thus allowing a labo- 
ratory user to completely control all stages of an individual call. 
Microprocessor-controlled operator simulators are provided to auto- 
matically handle calls at tsps operator positions. In addition, calls can 
be handled manually at an operator position, thereby allowing the 
system laboratory user to test unexpected operator sequences. 

System testing is divided into functional testing and system evalu- 
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ation. Ideally, it would be desirable to write and perform functional 
tests in the system laboratory for each call processing and maintenance 
situation the tsps will encounter. In practice, however, this is impos- 
sible. Consequently, during system testing, additional effort is required 
above and beyond the analysis of functional test results. This effort is 
referred to as system evaluation. In terms of cause and effect, func- 
tional testing involves setting up a prescribed set of initial conditions 
and then determining whether or not the proper response occurs. 
However, system evaluation involves observing every improper system 
response and then determining the cause of that response. This can be 
a difficult task, which at times is more of an art than a science. 
Reproducibility is a primary requirement for identifying the cause of 
a problem. Problems discovered during functional testing are generally 
reproducible, since a set of initial conditions have been specified for 
running each functional test. However, problems encountered during 
system evaluation frequently do not meet this reproducibility criterion. 
As a result, much more analysis of these types of problems is required. 
During system evaluation, several indicators are used to determine 
that a problem exists. These indicators are: 

(i) Unexpected teletypewriter output messages. 
(ii) Loss of service of a hardware unit to the system for no appar- 
ently valid reason. 
(Hi) Unexplainable maintenance or call processing activity. 

The audit messages 7 printed on the teletypewriter are a primary 
indication of system problems. Each audit message generally signifies 
the presence of some program error resulting in an inconsistency in 
unprotected memory. This memory is continuously updated by differ- 
ent programs to reflect the current state of the system. A detailed 
analysis of the audit messages will sometimes indicate what caused 
the error condition. The debugging capabilities of the minicomputer 
described above are particularly helpful in resolving this type of 
problem. With these capabilities, system activity which occurred be- 
fore the audit program detected the error can be analyzed, and the 
cause of the problem can be identified. 

The system laboratory provides a controlled environment where 
interfaces external to the tsps have been simulated with system 
utilities. Although it is possible to test most aspects of the tsps 
operation in this environment, increased confidence is built when new 
features are tested at a newly installed tsps which interfaces with 
actual local and toll offices. For these and other reasons, the verifica- 
tion and evaluation process is continued at a test site. 
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IV. THE SITE TESTING INTERVAL 

Each new generic is tested at a tsps that has not been cut into 
service. This tsps has been fully engineered by operating company 
personnel, and the hardware has been installed by Western Electric 
during a normal installation interval. Bell Laboratories testing of a 
generic at this pre-cutover tsps evaluates the new features in a fully 
equipped tsps. It also verifies the accuracy of the information provided 
to Western Electric and the operating companies on these new fea- 
tures. Testing is done with both local and toll offices to ensure that no 
interface problems exist. The length of the site test interval varies 
depending on the number of features being tested, amount and com- 
plexity of the new hardware, and size of the program change. Table I 
summarizes the test site, major new features, and size of the program 
change for each of the recent tsps generics. 

Testing at the site involves reverifying specific operational and 
maintenance capabilities for all new features, with the objective of 
detennining whether or not the requirements for each feature are met. 
In particular, testing focuses on verifying interfaces with local and toll 
offices. Regression testing is also performed to ensure that previous 
tsps capabilities are not adversely affected by the new features. Since 
site testing is performed after new feature development has been 
completed, problems identified at the test site are generally more 
subtle than those previously uncovered during system laboratory 
testing at Indian Hill. Furthermore, by the beginning of the site test 
interval, the hardware design has already been proven in the system 
laboratory. As a result, during this interval the majority of problems 
identified are in the software. Most are not due to any significant 
design problems and are easily corrected. 

Both functional testing and regression testing at the site are done in 
a systematic manner; a specific set of tests are performed for each 



Table I — Recent TSPS generics 



Generic 



Test Site 



Major New Feature(s) 



Size of Pro- 
gram Change 
(40-bit 
Words) 



Generic 7, Is- 
sue 1 

Generic 7, Is- 
sue 2 

Generic 8, Is- 
sue 1 

Generic 8, Is- 
sue 2 



Syracuse, New 
York 

Saginaw, Michi- 
gan 



Remote Trunk Arrangement, Po- 
sition Subsystem No. 2 

Selective Call Screening, more 
than twenty stores on a bus 



Phoenix, Arizona Automated Coin Toll Service 



Montgomery, Al- 
abama 



Automated Coin Toll Service with 
Remote Trunk Arrangement 



60,000 
5,000 

40,000 
7,000 
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feature, the results are observed, and these results are analyzed. The 
amount of detail with which functional test results are analyzed at the 
test site is minimal. Instead, any functional tests which fail are docu- 
mented in trouble reports which are then sent back to the developers 
at Indian Hill. Extensive analysis at the test site is restricted to 
problems uncovered during system evaluation. In general, the exact 
conditions required to bring out these types of problems must be 
determined at the test site. 

The combination of system laboratory testing and site testing during 
this interval complement each other in the verification and evaluation 
of a tsps generic. In the system laboratory at Indian Hill, emphasis is 
placed on testing individual changes being made to correct specific 
problems. However, as indicated above, effort at the test site is oriented 
toward verifying and evaluating functional capabilities rather than 
testing changes or corrections. 

The test facilities provided at the major test sites are comparable to 
those permanently installed in the system laboratories at Indian Hill. 
These facilities include the minicomputer, call generation capabilities, 
and operator simulators described in Section III. Additional capabili- 
ties are also provided to remotely control these test facilities so that 
testing can be done from one location. 

The number of problems identified from the time the hardware and 
software were frozen until the completion of site testing is shown in 
Table II for each of the recent tsps generics. These totals are broken 
down into: (i) problems identified at Indian Hill in the system labora- 
tory and (ii) problems identified at the test site. 

At the completion of the site testing interval, responsibility for 
monitoring system performance is given to tsps field support person- 
nel. In addition to continuing the evaluation of tsps and new feature 
performance, field support personnel are responsible for updating the 
system with any necessary changes. To assist in these efforts, the 
tsps's maintenance teletypewriter output can be transmitted to the 
tsps Diagnostic Center at Indian Hill. Problems detected from this 
output or reported by the operating company are analyzed, and cor- 

Table II — Trouble reports written through the completion 
of site testing 



Generic 



Generic 7, Issue 1 
Generic 7, Issue 2 
Generic 8, Issue 1 
Generic 8, Issue 2 



trs Written 






at the Base 


trs Written at 




Location 


the Test Site 
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600 


615 


1215 


33 


20 


53 


565 


800 


1365 


108 


58 


166 
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rections are generated by designers working in conjunction with field 
support personnel. 

V. SYSTEM CAPACITY 

The addition of each new feature to tsps has an impact on the real- 
time capacity of the tsps. Overall system real-time capacity varies for 
each tsps installation based upon the hardware configuration used 
and the particular call mix being processed. Each new feature is 
evaluated during the development cycle to determine its impact on 
system capacity. This evaluation is verified at the first in-service office 
or the first office close enough to capacity for a verification to be made. 
The real-time capacity estimation and verification process is extremely 
important due to its effect on the long-range planning of the operating 
companies. To assist the operating companies in their analysis of an 
individual office's real-time capacity, a program called tspscap is 
available. This program runs on an off-line computer and is updated 
with each generic to reflect the addition of new features. 

VI. SUMMARY 

System verification and evaluation is a process that is interwoven 
with all aspects of the development of new tsps features. From the 
inception of the idea for a new feature, development requirements are 
evaluated to ensure a proper understanding of the proposed capabili- 
ties and to verify that the proposed design will provide these capabil- 
ities. After formal reviews, these requirements are specified in devel- 
opment requirements memoranda, and the design is specified in de- 
velopment specifications memoranda and circuit descriptions. Next 
the design is implemented and each functional unit is verified in the 
system laboratory. Before the commencement of site testing, both 
hardware and software are placed in a frozen mode, after which all 
changes to the system are verified through a formal procedure. Site 
testing is done in a pre-cutover tsps engineered by an operating 
company with hardware supplied and installed by Western Electric. 
During site testing, functional tests are run to verify the proper 
implementation of all new features and regression tests are run to 
ensure the proper operation of previous tsps capabilities. In addition, 
system evaluation is done before the site is cut into service, thus 
resolving many of the more subtle problems. Finally, an analysis of 
the real-time effect of each new feature is made after cutover to 
confirm the theoretical analysis made prior to cutover. 

This verification and evaluation approach is followed for each new 
tsps feature. Adherence to this methodology allows new tsps generics 
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to be properly verified and evaluated, thereby insuring Bell System 
customers of the best possible service. 
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